创建数据目录描述

名称 目录条目元数据集

简介 用于集中存储与维护数据目录中各条目的标准化元数据，支撑数据发现、治理管控、权限管理、质量评估与合规审计。该数据集是目录服务的主数据源之一，提供条目级的业务与技术属性、血缘、分级与政策映射，确保目录信息一致、可追踪、可审计。

范围与用途

• 范围：覆盖目录内的全部可注册对象，包括数据集、表/视图、报表、指标、数据产品、管道/作业等条目。
• 用途：
  • 为数据发现提供可搜索的、结构化的条目元信息。
  • 为数据治理提供分级、责任人、质量评分、认证状态与合规标签。
  • 为安全与合规提供访问控制对象、审计线索与政策映射。
  • 为运营与生命周期管理提供变更记录、版本状态与退役计划。

数据结构（标准字段） 以下为建议的标准字段集合，实际实现可根据组织需要裁剪或扩展：

• entry_id（string）：目录条目唯一标识。主键，必须全局唯一。
• entry_name（string）：条目名称（规范命名）。
• entry_type（enum）：对象类型（dataset/table/view/report/metric/pipeline/data_product 等）。
• business_domain（string/enum）：所属业务域或主题域（来源于域目录）。
• business_description（string）：业务释义与用途摘要。
• technical_description（string）：技术说明（来源系统、结构要点、依赖）。
• glossary_terms（array[string]）：关联术语或业务定义（指向术语表）。
• source_system（string）：来源系统或系统记录（System of Record）。
• storage_location（string）：物理/逻辑存储位置（路径、库名、模式名）。
• lineage_upstream（array[string]）：上游依赖条目 ID 列表。
• lineage_downstream（array[string]）：下游消费条目 ID 列表。
• confidentiality_level（enum）：数据分级（公开/内部/受限/机密等）。
• pii_flag（boolean）：是否涉及个人敏感信息标识（针对条目内容，而非本元数据集）。
• regulatory_tags（array[string]）：适用的监管或合规标签（如隐私、财务、医疗等范畴）。
• data_owner（string）：数据所有者（角色或团队标识）。
• data_steward（string）：数据管理员/保管人。
• access_groups（array[string]）：可访问的角色/用户组（RBAC 对象）。
• certification_status（enum）：认证状态（未认证/已认证/受控/弃用）。
• dq_rules_applied（array[string]）：应用的数据质量规则 ID（规则库引用）。
• quality_score（number）：质量评分或等级（计算与定义需一致）。
• usage_metrics（json）：使用统计（浏览、查询、订阅等聚合指标）。
• lifecycle_stage（enum）：生命周期阶段（草稿/生产/弃用/退役）。
• retention_policy_id（string）：关联的保留与归档政策标识。
• change_request_id（string）：最近一次变更申请单号（变更管理系统）。
• created_at（datetime）：条目创建时间。
• updated_at（datetime）：最近更新时间。
• review_due_date（date）：下一次治理评审到期日。
• contact_channel（string）：咨询与支持渠道（通讯录或工单入口）。
• tags（array[string]）：自由标签（支持分类/检索）。

数据血缘与来源

• 血缘来源：通过编排平台、ETL/ELT、BI 工具与代码扫描自动采集；支持人工补充与校验。
• 元数据来源：目录服务、数据平台 API、规则库与权限系统；本数据集不载入实际数据内容，仅存储条目层的元信息与关联引用。

刷新与时效

• 更新模式：事件驱动（注册/变更触发）与定时同步（夜间批处理）结合。
• 时效要求：关键字段（分级、所有者、权限、认证状态、血缘）需在变更后定义窗口内同步；超时触发治理告警。

数据质量管理

• 完整性：entry_id、entry_name、entry_type、business_domain、data_owner、confidentiality_level 必填；非空校验。
• 一致性：entry_type、confidentiality_level 等采用受控词表；与域目录、术语表、规则库保持引用一致。
• 唯一性：entry_id 全局唯一；entry_name 在同一域内唯一（或提供去重策略）。
• 及时性：updated_at 不得早于来源变更事件指定阈值；超期记录纳入异常队列。
• 血缘完备度：lineage_upstream/lineage_downstream 对关键生产条目需覆盖率达标；缺失需标记与补录。
• 质量评分：quality_score由规则计算（如完整性、血缘完备度、权限配置正确率）；评分逻辑需在规则库中版本化。
• 监控与处置：对完整性、及时性、血缘缺失、分级不一致等设定监控；异常进入治理工单，按SOP分派数据管理员处理并留痕。

访问与安全

• 访问级别：本元数据集默认内部-受控；仅授权的治理、平台与审计角色具备读权限；写入通过目录服务与工作流。
• 授权模型：RBAC/ABAC结合；变更需工单审批与双人复核；审计日志记录读取与写入操作。
• 隐私注意：本数据集可能包含责任人/联系人标识（建议使用工号或群组标识，避免直接存储个人敏感信息）；如需邮箱/姓名，遵循最小化与遮蔽策略。

合规与政策映射

• 政策绑定：retention_policy_id、regulatory_tags 与组织的数据分级、保留与合规政策库保持引用一致。
• 审计支持：提供变更记录、认证状态与访问组信息以支持合规审计与证据留存。
• 数据最小化：仅存储条目级元信息与必要的治理属性，不载入或复制业务数据内容。

保留与生命周期

• 保留策略：随条目生命周期管理；退役条目元数据保留至政策规定期限，以满足审计与回溯。
• 归档与清理：达到保留期限后归档或清理；清理前需验证下游依赖与审计要求。

命名与版本管理

• 命名规范：统一前缀/域-对象-环境格式，避免歧义；entry_name 与存储位置保持可追溯映射。
• 版本控制：条目重大变更（结构、分级、所有者、认证状态）需版本化；变更原因、影响评估与审批记录入库。

角色与责任

• 数据所有者：对分级、用途与合规负责；审批访问与变更。
• 数据管理员/保管人：维护元数据完整性与准确性；处理质量异常与血缘补录。
• 目录平台团队：负责采集、同步、规则执行与审计支持。
• 合规/安全团队：定义政策与词表，监督执行与审计。

使用指引

• 注册与变更通过目录服务或工作流提交；直接写库仅限平台服务账户。
• 引用术语、域、规则、政策均使用标准 ID，避免自由文本造成不一致。
• 在项目上线、变更、退役时同步更新相关字段，确保血缘与权限实时有效。

风险与控制

• 风险：元数据缺失或滞后可能导致错误访问授权与合规风险。
• 控制：强制非空与受控词表校验、自动血缘采集与异常告警、变更审批与操作审计。

备注 本数据集为治理关键基础设施的一部分，任何结构调整、字段语义变更需提前评估对目录、权限、审计与质量计算的影响，并按变更管理流程执行。

Dataset name Conversion Funnel Events (转化漏斗事件集)

Summary Event-level dataset that standardizes and records user progression through defined conversion funnel stages across web and mobile channels. Designed for measuring funnel performance, diagnosing drop-off points, and supporting compliant attribution and optimization.

Business definition and scope

• Scope: Digital interactions that contribute to user progression from initial touch to conversion (e.g., purchase or signup). Includes on-site/app events and, where available, upstream acquisition signals from ad platforms.
• Use cases: Funnel conversion rate analysis, stage-to-stage drop-off, time-to-conversion, cohort progression, channel/campaign attribution, A/B test evaluation.
• Exclusions: System logs unrelated to user behavior, raw third-party network logs without consent, and any data lacking provenance or required metadata.

Grain

• One record per discrete event occurrence.
• Events are ordered by event_time within a user/session context.
• Canonical event_name values are versioned and governed (see “Event taxonomy”).

Event taxonomy (canonical funnel stages) Events are standardized to the following categories; implementation may include additional subtypes (versioned).

• Acquisition: ad_impression, ad_click
• Engagement: session_start, page_view, content_view, product_view
• Consideration: add_to_cart, wishlist_add
• Intent: checkout_start, payment_start
• Conversion: purchase_complete (order_placed), signup_complete (if applicable)
• Post-conversion: refund_requested, order_cancelled (optional, for lifecycle analysis)

Core schema (logical fields)

• Event identifiers
  • event_id: globally unique identifier (UUID)
  • event_name: canonical event type (enum; governed list)
  • event_version: schema version of the event payload
• Time
  • event_time: UTC timestamp of event occurrence
  • ingestion_time: UTC timestamp of data ingestion
• Identity and session
  • user_id: persistent authenticated identifier (nullable before login)
  • anonymous_id: device/browser-scoped identifier for unauthenticated users
  • session_id: server-side session identifier (governed sessionization rules)
• Source and context
  • platform: web, iOS, Android, other
  • device_type, os, app_version, browser
  • page_url / screen_name (web/app context)
  • geo_country, geo_region (derived; IP anonymized or truncated per policy)
• Marketing and attribution
  • source, medium, campaign_id, channel, term, content (UTM-standardized)
  • click_id (e.g., gclid/fbclid) where available
  • attribution_model (e.g., last_non_direct_click), attribution_window_days
• Commerce (if applicable)
  • product_id, sku, quantity, unit_price, currency
  • order_id, order_value, tax, shipping, discount_value
• Consent and privacy
  • consent_analytics, consent_advertising (boolean flags)
  • consent_timestamp, consent_source
  • data_subject_region (e.g., EU/US/Other)
• Quality and lineage metadata
  • source_system, pipeline_name, transform_version
  • is_duplicate (computed flag)
  • data_quality_score (composite metric; see checks below)

Keys and uniqueness

• Primary key: event_id (unique across the dataset).
• Secondary keys: (user_id, event_time) and (session_id, event_time) for ordered analysis.
• Deduplication: Exact dedup on event_id; domain-specific fuzzy dedup for purchases (e.g., identical order_id + user_id + order_value within a short window) per governed rule.

Ordering and sessionization

• Events are ordered by event_time within user_id or anonymous_id and session_id.
• Sessionization rules are defined and versioned (e.g., inactivity timeout, cross-tab continuity) and must be consistently applied across pipelines.

Identity resolution

• Identity stitching policy links anonymous_id to user_id upon authentication; cross-device resolution must use governed deterministic keys (e.g., login) or approved probabilistic methods with documented confidence.
• Identity graph changes are versioned; downstream consumers should not assume stability of historical joins without reprocessing windows.

Attribution rules

• Default reporting uses governed attribution_model and attribution_window_days. Changes require change control approval.
• Last-touch vs. first-touch and multi-touch models must be explicitly selected and documented in downstream reports.
• Direct traffic handling and bot/excluded traffic are defined by governance filters (see “Quality controls”).

Data quality controls

• Schema conformance: event_name must map to a governed schema; mandatory fields cannot be null (e.g., event_id, event_time, platform).
• Primary key uniqueness: no duplicate event_id.
• Referential integrity: product_id and order_id must exist in mastered product/order dimensions when present.
• Timeliness: data availability meets defined SLA (e.g., T+1 for analytics; near-real-time for operational use if applicable).
• Volume and distribution checks: anomaly detection on event counts by platform, channel, and stage.
• Reasonableness checks: monotonic funnel progression rates within expected bounds; spike detection on conversion events.
• Bot/invalid traffic filters: governed lists and heuristics applied consistently (e.g., known bot UA, data center IPs).

Lineage and sources

• Source systems may include client-side SDKs, server-side event collection, commerce backend, and ad tech platforms (distinctly tagged via source_system).
• Transformations: normalization, enrichment (UTM parsing, geo derivation), identity stitching, deduplication, and schema versioning recorded in transform_version metadata.

Refresh cadence

• Batch ingestion and/or streaming pipelines depending on source. Cadence is documented per source_system and pipeline_name.
• Late-arriving data and backfill policy are governed; reprocessing windows are documented in change logs.

Privacy and compliance

• Classification: Contains personal data; treat as Confidential/Restricted per Data Classification Policy.
• Legal basis: Analytics processing must respect consent_analytics; advertising use must respect consent_advertising, applicable regulations (e.g., GDPR/UK GDPR/CPRA), and internal policy.
• Data minimization: Store only necessary identifiers; IP must be anonymized/truncated per policy.
• Data subject rights: Deletion requests cascade to this dataset; lineage tags must support discoverability and purge.
• Cross-border transfer: Follow organizational data transfer controls; ensure appropriate safeguards for EU data where applicable.

Access and usage restrictions

• Access is role-based with least privilege. Raw event-level identifiers are restricted; use aggregated views for broad access.
• Prohibited: Re-identification, combining with external datasets without approved DPIA/assessment, individual-level decisioning without appropriate legal basis and approvals.

Retention and purging

• Retention follows the Data Retention Policy; raw events retained only as long as necessary for stated purposes. Aggregated derivatives may have different retention.
• Legal hold and deletion workflows are supported via lineage and subject resolution.

Stewardship and ownership

• Data Owner: Designated business owner (e.g., Growth/Marketing Analytics).
• Data Steward: Appointed steward responsible for policy conformance, metadata completeness, and change control.
• Technical Owner: Data engineering team responsible for pipelines and quality enforcement.

Change management

• Breaking changes (schema, event_name taxonomy, sessionization rules, attribution defaults) require governance review, version increment (event_version), deprecation schedule, and communication to consumers.
• Backward-compatible changes are documented and tagged; downstream consumers should validate impact.

Known limitations

• Ad impression completeness and deduplication vary by partner capabilities and consent.
• Cross-device identity stitching may be incomplete, affecting multi-touch attribution.
• Offline conversions are included only if integrated and consented; otherwise, conversion metrics reflect digital channels.

Usage guidance

• Compute funnel metrics by ordering events within a consistent identity and session scope; avoid mixing user_id and anonymous_id without applying the identity stitching policy.
• Use governed attribution fields rather than custom logic to ensure consistency across reports.
• For regulatory compliance, always filter analyses by consent flags when the purpose involves analytics or advertising.