Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Data governance: catalogs, lineage & classification

Why two correct dashboards can disagree, and what a data catalog records to settle it: owners, certified datasets, lineage traced upstream and downstream, sensitivity tags that drive masking and grants, and the retention rules those tags feed.

Which number is the real one?

"Which one is right?" Someone asks it whenever two dashboards that read the same warehouse disagree, and the first guess is nearly always a bug in one of them. At a car insurer, one dashboard showed 41,200 active policies at the end of the quarter and the other 38,900. It took an analyst three days to find out why: the first is built on a table called policies_clean, the second on policy_snapshot_v2, and the second left out policies in their 30-day grace period after a missed payment. Nobody knew which table was the official one, who had built either, or that the second existed.

There was no bug. The pipelines ran, the data was correct, and each dashboard queried what it was told to; what was missing was knowledge about the data. As data spreads across warehouses, lakes and pipelines, teams lose track of what exists and whether it can be trusted. Data governance is the practice of deciding who answers for each dataset and which rules it follows, and of keeping that knowledge: what each dataset is, who owns it, where it came from, how sensitive it is and how long it may be kept. Most of it ends up in one system.

A search box for the data estate

A data catalog (DataHub, Glue Data Catalog, Collibra) indexes datasets with their schema, owner and lineage, the upstream sources and transformations, so analysts can discover data and trace a number back to its origin. It holds metadata, not the data: table and column names, types, descriptions, freshness, who queries what. Connectors harvest most of it on a schedule from the warehouse, the lake, the transformation jobs and the dashboards. That matters, because a catalog filled in by hand starts going stale as soon as it is published.

People add what no connector can know: what a row means, and which table to trust. At the insurer, the finance team marks policies_clean as the certified source for active policies (each catalog has its own word for this) and writes in its description that grace-period policies count. policy_snapshot_v2 is marked deprecated, with a pointer to the certified table. The next analyst who searches for active policies finds both and knows which to use. That only works if the catalog is what analysts actually check before trusting a number; treated as optional documentation, it is the first thing nobody updates.

Someone to ask

Each dataset in the catalog names an owner: the team that answers questions about it, decides who may read it and announces changes to it. Even with one, a change can break things quietly. The claims team starts writing lapses to a new column, lapsed_at, and leaves the old lapse_date empty for new rows. Every nightly load succeeds. The policy dashboard's grace-period filter still reads lapse_date and treats an empty one as not lapsed, so from that day the dashboard counts no new lapses and wrongly keeps the lapsed policies in its active count for weeks.

A dataset with no listed owner leaves its consumers with nobody to ask when a schema change breaks them. Here the owner existed and still could not have warned anyone, because nothing told the claims team who read their column.

Following a number up, and a change down

Lineage is that missing record. It says the dashboard reads policies_clean, which a dbt model builds from raw.policies and raw.payments, which a nightly job copies from the policy system. It is a graph, and it is read in both directions:

  • Upstream, to explain a number. The 41,200 traces back through one model to two raw tables, the 38,900 through a different model with an extra filter. Three days of digging becomes a comparison of two paths and a read of the one model that differs.
  • Downstream, before a change. Before the claims team stops filling lapse_date, lineage lists every model and dashboard that reads it, and their owners.
Catalog indexes the data estate

Each layer reports its schema and lineage to the catalog, where owners, certification and sensitivity tags are added. The warehouse enforces the policies attached to those tags.

Most of the graph is collected automatically, from the jobs that build each table and the queries the warehouse runs. Column-level lineage goes a step further and says which source columns feed each output column. The lapse_date question needs that, including columns a model only filters on, which not every tool records.

Lineage covers only the jobs that report it. A notebook that exports to a spreadsheet, or a script on someone's laptop that writes back to the lake, leaves no edge, and the graph looks complete when it is not. With no lineage at all, tracing a questioned number back to its source means rebuilding the path by hand, as the analyst's three days did. ETL/ELT & data orchestration explains how the pipelines are built and scheduled, and Analytics data: OLTP vs OLAP & the warehouse the warehouse they feed.

Tags that change what a query returns

The policies table also holds date_of_birth, licence_number and, for claims, injury_description. The catalog gives each column a classification, a sensitivity tag such as pii or pii.health. Scanners help find them: services such as Google Cloud DLP and Amazon Macie sample stored data and flag what looks like a national ID or a phone number, and the owner confirms the tag. That catches what a review by column name misses, such as a free-text notes column full of phone numbers.

A tag on its own is only a label; it earns its place when access policies key off it. In warehouses such as BigQuery and Snowflake, a masking or access policy can be attached to the tag rather than written column by column: analysts querying date_of_birth see a masked value, the fraud team's role sees the real one, and a new column that receives the tag gets the same rule without anyone writing a new grant. Where tags are the only thing that applies those policies, PII datasets with no sensitivity tags never get those access policies (column masking, grants) applied, and anyone who can read the table can query the PII through the warehouse.

How long it may stay

Catalog classes also drive retention rules, by kind of data as well as sensitivity. The insurer's policy keeps raw telematics from customers' cars for 90 days, and claims records for as long as each country's law requires. The retention job does not work table by table: it asks the catalog for every dataset tagged as raw telematics and deletes what is older than the limit, including the copy an analyst made last month, provided that copy was registered and tagged. Retention is only as complete as the catalog's map of where each class of data lives.

The lakehouse, and what comes next

The Analytics Lakehouse challenge is where this is practised: one of its requirements is to catalog datasets with owners, schemas and lineage. The task is to connect the Data Catalog to every layer that produces a dataset, with what each one reports written on its edge, to give each layer an owner, and to leave one dashboard number traceable back to a raw table. Tagging one sensitive raw column, with the warehouse policy that enforces it, is an optional extra.

A request to delete one customer's data starts from the same catalog, which says which stores hold customers and under which key: a store the catalog does not know about keeps the data. Deleting a person's data: erasure across every copy picks up from there.

Practice this