制定数据质量目标

Data Quality Goals for User Feedback Text

Scope

• Applies to free-form user feedback text and its associated metadata (e.g., timestamp, channel, user identifier, language, product area).
• Goals cover ingestion, cleansing, validation, labeling, storage, and monitoring of the dataset used for analytics and model training.

Quality Objectives, Metrics, and Targets

1. Schema Validity

• Goal: All records conform to a defined schema and field types.
• Metric: Validity rate = valid_records / total_records.
• Target: ≥ 99.9% validity; ≤ 0.1% rejected at ingestion.

2. Required Field Completeness

• Goal: Mandatory fields present and populated (e.g., timestamp, source channel, language code, product area).
• Metric: Completeness rate = records with all required fields / total_records.
• Target: ≥ 99.5%.

3. Content Presence and Encoding

• Goal: Text field non-empty and properly encoded.
• Metrics:
  • Non-empty content rate = records with non-empty, non-whitespace text / total_records; Target ≥ 99%.
  • UTF-8 compliance rate = utf8_valid_text / total_text; Target 100%.

4. Normalization and Consistency

• Goal: Canonical representation for language codes, punctuation, whitespace, and URLs/emojis.
• Metrics:
  • ISO 639-1 language code compliance = records with valid codes / total_records; Target 100%.
  • Normalization applied rate (trim, collapse whitespace, strip HTML, standardize URLs to placeholders, normalize emojis) = normalized_records / total_records; Target ≥ 99.5%.

5. Deduplication and Near-Duplicate Control

• Goal: Remove exact and near-duplicate feedback while retaining canonical versions.
• Metrics:
  • Residual duplicate rate after dedup = duplicates_remaining / total_records; Target ≤ 0.5%.
  • Near-duplicate cluster coverage = records in dedup clusters / estimated duplicates; Target ≥ 95%.
• Method: Hash-based exact match + text similarity (e.g., MinHash/Jaccard or cosine on embeddings) with validated thresholds.

6. Language Identification Quality

• Goal: Accurate language detection with fallback to “undetermined” when confidence is low.
• Metrics:
  • Coverage = records with language assigned / total_records; Target ≥ 99%.
  • Accuracy on benchmark labeled set = correct_language / benchmark_total; Target ≥ 95%.
  • Undetermined rate ≤ 2% (except channels known for code-mixing).
• Controls: Confidence score min threshold; second-pass model for short texts.

7. Spam/Bot Filtering

• Goal: Exclude spam, promotional content, and automated/bot feedback.
• Metrics (evaluated on labeled QA sets):
  • Precision ≥ 98%; Recall ≥ 90%; False positive rate ≤ 1%.
• Controls: Multi-signal detection (rate limits, token patterns, URLs, repetition, account heuristics).

8. PII Handling and Safety

• Goal: Detect and mask PII; prevent unmasked PII in curated datasets; label toxic content.
• Metrics:
  • PII masking completeness (recall on labeled set) ≥ 95%.
  • Unmasked PII occurrence ≤ 0 per curated dataset release.
  • Harmful/toxic content false negative rate ≤ 2% on safety benchmark.
• Controls: Pattern rules + ML NER; irreversible masking for storage; access controls and audit logs.

9. Sentiment/Category Label Quality (if annotated)

• Goal: Reliable labels for sentiment, issue type, priority, or intent.
• Metrics:
  • Inter-annotator agreement (Cohen’s κ) ≥ 0.80 on overlapping samples.
  • Gold set accuracy ≥ 90%.
  • Label completeness ≥ 99% for records in labeled subsets.
• Controls: Clear labeling guidelines; ongoing calibration; adjudication of disagreements.

10. Metadata Integrity and Referential Consistency

• Goal: Stable linkages between feedback and entities (user/account, product, release/version).
• Metrics:
  • Referential integrity rate = records with valid foreign keys / total_records; Target ≥ 99.9%.
  • Orphan record rate ≤ 0.1%.
• Controls: FK constraints (where feasible), dimension conformance checks.

11. Timeliness and Availability

• Goal: Low-latency ingestion and high data availability.
• Metrics:
  • Ingestion latency p95 ≤ 5 minutes; p99 ≤ 15 minutes from event to warehouse availability.
  • Pipeline availability ≥ 99.9% monthly.
• Controls: Backfill procedures; retry with dead-letter queue; SLA monitoring.

12. Provenance and Lineage

• Goal: Full traceability of source, transformations, and versions.
• Metrics:
  • Provenance completeness (source, ingest time, pipeline version) = records with provenance / total_records; Target 100%.
  • Versioned dataset releases with changelog coverage 100%.
• Controls: Data catalog entries; immutable audit logs.

13. Readability and Noise Reduction

• Goal: Remove artifacts that hinder NLP and analysis.
• Metrics:
  • HTML/markup removal rate ≥ 99.5%.
  • URL standardization rate ≥ 99.5% (e.g., replace with token).
  • Non-language noise reduction rate (ASCII art, long repeats) ≥ 95%.
• Controls: Deterministic cleaners with unit tests; language-aware tokenization.

14. Numeric/Text Validation for Structured Fields Embedded in Feedback

• Goal: Validate embedded IDs, order numbers, or version strings when present.
• Metrics:
  • Format validity ≥ 99%; Cross-system reconciliation accuracy ≥ 98% on sampled matches.
• Controls: Regex + checksum rules; lookup validations.

15. Observability and Quality Monitoring

• Goal: Continuous detection and response to quality degradations.
• Metrics:
  • Daily quality job success rate ≥ 99.9%.
  • Alerting time-to-detect ≤ 15 minutes for threshold breaches.
  • QA sampling coverage ≥ 1% of daily volume or ≥ 500 records, whichever is greater.
• Controls: Dashboards per metric; automated alerts; incident runbooks; weekly quality review.

Measurement and Validation Methods

• Baseline: Establish initial metrics with a two-week historical audit; set thresholds as above and adjust once stabilized.
• Sampling: Stratified sampling across channels, languages, and lengths; include edge cases (very short, emoji-heavy).
• Benchmarks: Maintain labeled gold sets for language, spam, PII, toxicity, and sentiment; refresh quarterly.
• Formulas: Define metric formulas in the data catalog with owners and calculation code paths; version and test them.

Acceptance Criteria

• A dataset release is approved only if all critical goals (1–5, 7–12, 15) meet targets, and non-critical goals (6, 13, 14) meet minimum thresholds within 95% of target.
• Any breach triggers incident response: root cause analysis within 24 hours; corrective action plan and revalidation before publication.

Ownership

• Metric owners per domain (Schema/ETL, NLP/Labeling, Security/Privacy, Observability) with documented on-call rotation and escalation paths.

These goals provide measurable, enforceable standards to ensure user feedback text is accurate, complete, consistent, secure, and reliable for downstream analytics and machine learning.

以下為「需求文檔集」的數據質量目標與量化指標，採用可度量（SMART）原則，聚焦完整性、一致性、正確性、時效、追蹤性、可讀性、可用性、安全合規與可維護性等維度。適用對象包含：BRD、FRD、用戶故事、非功能需求、介面規格、需求追蹤矩陣、變更請求與決議記錄。

一、範圍與定義

• 文檔層級：每份文檔的元資料、版本與批准狀態。
• 需求層級：每條需求的唯一標識、描述、驗收準則、來源、優先級、依賴/約束與上下游追蹤鏈。
• 資料形態：結構化欄位（ID、狀態、優先級等）與非結構文本（描述、理由、準則等）。

二、質量目標與量化指標

1. 完整性

• 文檔元資料完整率≥98%（必填：文檔ID、版本、狀態、作者、審核人、批准日期、需求來源、優先級、系統範圍）。
• 需求欄位完整率≥95%（必填：需求ID、描述、業務理由/價值、驗收準則、來源、優先級、依賴/約束、狀態、追蹤鏈接）。

2. 一致性

• 詞彙標準化合規率≥98%（使用受控詞彙與命名規範，禁止同義詞混用）。
• 結構與編碼一致率≥99%（章節編號、版式、欄位字典一致）。
• 重複/矛盾需求未解決數量=0（自動檢出＋人工複核；提出後≤5個工作日內關閉）。

3. 正確性與有效性

• 來源對齊正確率≥98%（需求與批准的業務目標/變更決議一致，具可驗證證據）。
• 可測試性覆蓋率≥95%（每條需求具明確、可驗證的驗收準則）。
• 範圍合規：未授權範圍外需求比例=0。

4. 時效性

• 變更至文檔更新的中位週期≤2工作日；95百分位≤5工作日。
• 文檔提交至完成審核的週期≤5工作日。

5. 追蹤性（可溯源）

• 上游覆蓋率：需求→業務目標=100%。
• 下游覆蓋率：需求→測試用例≥95%；需求→設計工件≥90%。
• 版本與審批記錄完備率=100%（變更原因、日期、責任人、決議）。

6. 可讀性與無歧義

• 含模糊/主觀用語比例≤2%（如「快速」、「易用」、「最好」、「適當」等未具體量化）。
• 強制用語規範合規率≥98%（MUST/SHALL/SHOULD等使用一致並可驗證）。

7. 可用性與可取得性

• 權限配置正確率=100%（最小權限、審批控制）。
• 最新版本可達成性：授權用戶在≤3次操作取得；文檔鏈接有效率≥99%。

8. 安全與合規

• 敏感信息標記/遮蔽合規率=100%（個資、商業機密）。
• 存儲與傳輸加密合規率=100%；保留與銷毀政策符合既定標準。

9. 可維護性與清潔度

• 陳舊文檔標記率=100%；過時文檔清理週期≤30天。
• 冗餘/重複段落比例≤3%（與其他文檔/需求重複者合併或關閉）。

10. 下游影響控制

• 因需求文檔質量缺陷導致的測試缺陷比例≤5%。
• UAT返工中歸因「需求不清/不一致」比例≤3%。

三、指標計算與驗證方法

• 自動化校驗：在版本庫/需求管理工具中運行規則引擎與靜態語言檢查（受控詞彙、格式、必填欄位、模糊詞彙、重複/矛盾偵測）。
• 抽樣稽核：每月對≥10%文檔與≥10%需求進行人工審核，雙人複核一致性≥95%。
• 追蹤鏈一致性檢查：比對需求ID與上/下游關聯表；斷鏈即告警。
• 週期監控：儀表板按日更新KPI；例外自動告警與工單化追蹤。

四、治理與責任

• 角色分工：
  • 需求管理員：維持欄位完整、一致性與追蹤鏈覆蓋。
  • 資料品質負責人（DQA）：制定/維護規則與KPI，監控與稽核。
  • 變更控制委員會（CCB）：審批重大變更與例外。
  • 測試與架構代表：驗收「可測試性」與「設計對齊」。
• 流程要求：所有變更需關聯工單與決議；未通過自動化檢查的提交不得合併。

五、例外管理與持續改進

• 例外申請：需提供業務理由、暫行風險控制與糾正計畫；有效期≤30天。
• 根因分析：對未達標指標進行月度RCA，制定改進措施與時間表。
• 成功標準：連續兩個季度達成全部閾值；KPI告警率<5%；下游缺陷持續下降趨勢。

上述目標旨在確保需求文檔集在全生命週期中可依賴、可驗證、可追蹤，並將對下游設計、測試與交付的質量風險降至可控水平。