Lineage Depth and Coverage
Description
This recipe assesses the depth and completeness of column-level lineage recorded in the catalog for every active table. It classifies each asset by lineage presence (Connected, Upstream Only, Downstream Only, or Orphan), calculates what percentage of its active columns actually participate in a lineage edge, and traces backward up to four hops to determine how far its data can be reliably sourced. Cycle-safe traversal ensures depth results remain accurate even in circular lineage graphs. The output flags orphan and low-coverage assets that need lineage remediation, and lays the groundwork for future impact-analysis capabilities.
Details
Most tables in the catalog appear to have lineage, but no one currently knows how deep, how complete, or at what level of detail that lineage actually is. This recipe closes that gap without assuming column-level lineage exists — it identifies orphan assets with zero lineage, distinguishes genuine column-to-column mappings from table-level and file/chart-sourced evidence, and traces backward — safely, even through circular lineage graphs — to determine exactly how many hops back a table's data can be reliably sourced. The output is a clear, asset-level view of lineage depth and traceability across every connected system, with a readiness label that never overstates the grain of evidence actually available. It forms the foundational signal that future impact-analysis capabilities will build on.
Business value
- Surfaces orphan assets with no lineage that were previously invisible
- Distinguishes genuine column-level lineage from table-level and file/chart-sourced evidence, instead of assuming all lineage is equally granular
- Correctly separates columnless tables from tables with real coverage gaps, avoiding misleading 0% readings
- Establishes how far back each table can be reliably traced, bucketed for quick scanning
- Detects circular lineage paths so depth results are never inflated or misleading
- Creates the foundational readiness signal that future impact-analysis recipes will consume
Who benefits
- Data governance leads tracking lineage documentation gaps across connections
- Data engineers deciding where to prioritize lineage remediation work
- Platform/catalog admins reporting lineage health and grain quality by connection and schema
- Stakeholders evaluating which assets are safe to build downstream impact analysis on, and at what level of detail
What you receive
How the recipe works (guided flow)
Step 1 — Build Active Asset Inventory
Compiles every active table and column in scope as the full population, respecting both table-level and schema-level active status, so no asset is ever silently dropped before analysis begins.
Step 2 — Resolve Lineage Edges Across All Grains
Combines table-level relationships from the catalog's lineage table with type-aware resolution of column-level, table-level, and external file/chart-sourced records, so no genuine relationship is missed just because it wasn't captured at column granularity.
Step 3 — Classify Lineage Presence
Tags every asset as Connected, Upstream Only, Downstream Only, or Complete Orphan, based on all resolved evidence — including relationships that trace to a file or chart rather than another table.
Step 4 — Calculate Grain-Aware Coverage
Determines, for each table's upstream lineage specifically, whether the evidence is genuine column-level mapping, table-level only, or file/chart-sourced only — and separately flags tables with no columns cataloged, so a lack of columns is never confused with a lack of lineage.
Step 5 — Trace Lineage Depth Backward
Walks backward from each table up to four hops to determine how far its data can be traced to a source, correctly counting a file or chart origin as one real hop, with built-in cycle detection to keep results accurate.
Step 6 — Determine Lineage Readiness Signal
Combines presence, grain-aware coverage, and depth into one honest readiness label per asset — distinguishing table-level, file/chart-level, and column-level traceability — along with a plain-English explanation of any gap.
Step 7 — Summarize and Visualize Results
Rolls up readiness and depth results into summary tables and charts by connection, schema, and asset type for quick reporting.
Sample insights (illustrative)
| Insight Category | What the recipe discovered | Business Impact |
|---|---|---|
| Lineage Grain Reality Check | Across the Snowflake_Prod connection, over 95% of recorded lineage is table-level or file/chart-sourced — almost none is genuine column-to-column mapping | Prevents teams from assuming column-level traceability exists when it doesn't, avoiding false confidence in downstream impact analysis |
| Orphan Detection | 18% of active tables in the Snowflake_Prod connection have zero recorded lineage in any direction or grain | Flags previously invisible blind spots before any impact analysis can rely on this data |
| Depth & Cycle Detection | A finance reporting table traces back 3 hops to its source, with one circular path safely detected and excluded | Gives an accurate depth measure teams can trust instead of an inflated or misleading trace |
Make sure the following ingredients are available in your workspace:
- Connected datasets are crawled and profiled
- Columns have distinct counts and top values generated