{"slug": "securing-golden-data-for-ai-governance-metrics-on-databricks", "title": "Securing Golden Data for AI Governance Metrics on Databricks", "summary": "A developer outlined a set of Unity Catalog controls for securing the golden data behind AI governance dashboards on Databricks, including a dedicated schema with restricted MODIFY grants, lineage queries against system.access.table_lineage to catch eval-set leakage into prompts or retrieval indexes, audit-log monitoring of permission changes, and Delta time-travel versioning so every reported metric can be tied to a specific data version. The approach treats the number of principals able to edit the answer key as a governance metric itself, and flags reads from interactive notebooks or copies to tables outside the evaluation results schema as warning signs.", "body_md": "AI governance dashboards report accuracy, groundedness and safety scores. Every one of those scores is a comparison against golden data: the eval sets, expected answers and reference data your team has agreed to treat as truth.\n\nIf that data can be quietly edited, leaked into prompts or lost to retention, the dashboard still shows numbers. They just stop meaning anything.\n\nThis post walks through the controls I use to secure golden data on Databricks, with SQL you can adapt. Everything here uses native Unity Catalog features.\n\nPut golden data in a dedicated schema so you can govern it as one unit. Readers get SELECT. A small owners group gets MODIFY.\n\n```\nCREATE SCHEMA IF NOT EXISTS ai_gov.golden\n  COMMENT 'Golden eval sets and reference data. Changes need review.';\n\nGRANT USE CATALOG ON CATALOG ai_gov TO `eval-readers`;\nGRANT USE SCHEMA, SELECT ON SCHEMA ai_gov.golden TO `eval-readers`;\n\nGRANT USE CATALOG ON CATALOG ai_gov TO `golden-data-owners`;\nGRANT USE SCHEMA, SELECT, MODIFY ON SCHEMA ai_gov.golden TO `golden-data-owners`;\n```\n\nThen measure it. The number of principals that can change the answer key is a governance metric in its own right.\n\n```\nSELECT grantee, privilege_type\nFROM system.information_schema.schema_privileges\nWHERE catalog_name = 'ai_gov'\n  AND schema_name = 'golden'\n  AND privilege_type IN ('MODIFY', 'ALL PRIVILEGES');\n```\n\nThis query only covers schema-level grants. Also check table-level grants, catalog-level grants and object owners, because owners and admins can change data too.\n\nLeakage is the quiet killer. Someone pastes eval questions into a prompt, or a pipeline copies the set into a retrieval index. Scores go up, and the model has effectively seen the answers.\n\nUnity Catalog writes table lineage to `system.access.table_lineage`, kept for 395 days by default. Each row records the entity (notebook, job, pipeline, query or dashboard), who generated it, and the source and target. A row with a source and no target is a read.\n\nThis query lists every read of the golden schema in the last 30 days by anyone other than your evaluation service principal:\n\n```\nSELECT event_time,\n       created_by,\n       entity_type,\n       entity_id,\n       source_table_full_name,\n       target_table_full_name\nFROM system.access.table_lineage\nWHERE source_table_catalog = 'ai_gov'\n  AND source_table_schema = 'golden'\n  AND event_date >= date_sub(current_date(), 30)\n  AND coalesce(created_by, '') <> '<your-eval-service-principal-id>'\nORDER BY event_time DESC;\n```\n\nTwo rows deserve special attention: any read from an interactive notebook, and any row where `target_table_full_name` is outside your evaluation results schema. The second one means the golden data was copied somewhere.\n\nRun it as a scheduled query with an alert. One caveat from the docs: lineage is only recorded when it can be inferred from the event, so treat this as strong evidence, not a perfect seal.\n\nGrants drift over time. Someone adds a group \"temporarily\" and never removes it. The audit system table, `system.access.audit`, keeps 395 days and records Unity Catalog actions.\n\n```\nSELECT event_time,\n       user_identity.email AS changed_by,\n       action_name,\n       request_params\nFROM system.access.audit\nWHERE service_name = 'unityCatalog'\n  AND action_name = 'updatePermissions'\n  AND event_date >= date_sub(current_date(), 30)\n  AND to_json(request_params) LIKE '%ai_gov.golden%'\nORDER BY event_time DESC;\n```\n\nCheck the exact action names and request parameter keys in the audit log reference for your cloud before you depend on this.\n\nA score without a data version can't be reproduced. Delta records every version of a table:\n\n```\nDESCRIBE HISTORY ai_gov.golden.qa_eval;\n```\n\nHave your evaluation job read a specific version and write that version number next to every metric it produces:\n\n```\nSELECT * FROM ai_gov.golden.qa_eval VERSION AS OF 42;\n```\n\nWatch the retention trap. By default, Delta keeps 30 days of table history (`delta.logRetentionDuration`) but only 7 days of removed data files (` delta.deletedFileRetentionDuration`), and `VACUUM` deletes older files. After that, older versions can't be queried. If an auditor asks you to reproduce a number from March, time travel won't help.\n\nSo freeze a snapshot for each release:\n\n```\nCREATE TABLE ai_gov.golden_releases.qa_eval_2026_11\n  DEEP CLONE ai_gov.golden.qa_eval VERSION AS OF 42;\n```\n\nGrant SELECT only on the releases schema, and don't let anyone modify it.\n\nGolden sets built from real cases often carry personal or health data. The tempting fix is a \"safe\" copy, which then drifts from the real one.\n\nAttribute-based access control (ABAC), governed tags and automated data classification in Unity Catalog have been generally available since May 2026. Classification can flag likely sensitive columns. You tag them, and one policy masks every tagged column in the schema:\n\n```\nCREATE FUNCTION ai_gov.policies.redact(v STRING)\n  RETURNS STRING\n  RETURN '[REDACTED]';\n\nCREATE POLICY mask_sensitive_in_golden\nON SCHEMA ai_gov.golden\nCOMMENT 'Mask tagged sensitive columns for everyone except owners'\nCOLUMN MASK ai_gov.policies.redact\nTO `account users` EXCEPT `golden-data-owners`\nFOR TABLES\nMATCH COLUMNS has_tag('sensitive') AS sensitive_col\nON COLUMN sensitive_col;\n```\n\nNew tables in the schema are covered as soon as their columns carry the tag. Reviewers can still label cases. The raw values never leave the platform.\n\n**Freshness.** Truth changes, and a golden set nobody has touched in a year is probably wrong somewhere. Data quality monitoring (formerly Lakehouse Monitoring) can learn each table's normal commit timing and row volume and mark it stale or incomplete when that breaks. Results land in `system.data_quality_monitoring.table_results`.\n\n**The judge.** If you grade with an LLM, use a different model family from the one you're grading. Research presented at NeurIPS 2024 showed LLM evaluators can recognize and favor their own outputs. Check judge agreement with human labels on a sample every release. MLflow 3 stores evaluation datasets as Unity Catalog tables and records the source of each record, a person, a document or a trace, which helps when someone challenges an expected answer.\n\n**One definition per metric.** If groundedness is computed three ways in three notebooks, people will quote the one they like. Unity Catalog metric views let you define a measure once and reuse it across dashboards.\n\nThe final step is to make all of this visible next to the model metrics:\n\n| Metric | Source | \n|---|---|\n| Principals who can write to the golden schema | information_schema privileges, plus owners | \n| Reads of golden data outside evaluation jobs (30 days) | system.access.table_lineage | \n| Permission changes on the golden schema (30 days) | system.access.audit | \n| Golden sets past their review date | your review log, data quality monitoring | \n| Sensitive columns covered by a masking policy | governed tags and policies | \n| Judge agreement with human labels | your evaluation runs | \n\nWhen these look healthy, people trust the model metrics beside them. When they don't, you find out before an auditor does.\n\nThe EU AI Act asks providers of high-risk systems to govern their training, validation and testing data (Article 10). The AI Omnibus, published in July 2026, moved the Annex III high-risk deadline to December 2, 2027. That's time to build this properly, not a reason to skip it.\n\nI'm Anil Prasad. I lead software, data and AI engineering in healthcare, life sciences, energy and finance, with a focus on AI that holds up in regulated environments. I write Field Notes: Production AI on LinkedIn and post here about the engineering behind it.\n\nanilsprasad.com | linkedin.com/in/anilsprasad | github.com/anilatambharii", "url": "https://wpnews.pro/news/securing-golden-data-for-ai-governance-metrics-on-databricks", "canonical_source": "https://dev.to/anilatambharii/securing-golden-data-for-ai-governance-metrics-on-databricks-4o10", "published_at": "2026-11-20 21:09:43+00:00", "updated_at": "2026-11-20 21:14:48.068965+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "mlops", "ai-infrastructure", "large-language-models"], "entities": ["Databricks", "Unity Catalog", "Delta Lake"], "also_reported_by": [], "takeaway": "Restrict MODIFY on the golden eval schema to a small owners group, then schedule lineage and audit queries against system.access.table_lineage and system.access.audit to catch eval-set reads outside the evaluation service principal and permission drift.", "recommendation": {"advice": "Version-pin golden eval sets with Delta time travel and record the version beside every metric, and alert on lineage reads of the golden schema by anyone but the evaluation service principal.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/securing-golden-data-for-ai-governance-metrics-on-databricks", "markdown": "https://wpnews.pro/news/securing-golden-data-for-ai-governance-metrics-on-databricks.md", "text": "https://wpnews.pro/news/securing-golden-data-for-ai-governance-metrics-on-databricks.txt", "jsonld": "https://wpnews.pro/news/securing-golden-data-for-ai-governance-metrics-on-databricks.jsonld"}}