Lineage Depth and Coverage
Description
This recipe assesses the depth and completeness of column-level lineage recorded in the catalog for every active table. It classifies each asset by lineage presence (Connected, Upstream Only, Downstream Only, or Orphan), calculates what percentage of its active columns actually participate in a lineage edge, and traces backward up to four hops to determine how far its data can be reliably sourced. Cycle-safe traversal ensures depth results remain accurate even in circular lineage graphs. The output flags orphan and low-coverage assets that need lineage remediation, and lays the groundwork for future impact-analysis capabilities.
Details
Most tables in the catalog appear to have lineage, but no one currently knows how deep or how complete that lineage actually is. This recipe closes that gap by identifying orphan assets with zero lineage, quantifying how much of each table's data is truly traceable, and tracing backward — safely, even through circular lineage graphs — to determine exactly how many hops back a table's data can be reliably sourced. The output is a clear, asset-level view of lineage depth and coverage across every connected system, forming the foundational signal that future impact-analysis capabilities will build on.
Business value
- Surfaces orphan assets with no lineage that were previously invisible
- Quantifies column-level lineage coverage instead of a vague "connected / not connected" flag
- Establishes how far back each table can be reliably traced, bucketed for quick scanning
- Detects circular lineage paths so depth results are never inflated or misleading
- Creates the foundational readiness signal that future impact-analysis recipes will consume
Who benefits
- Data governance leads tracking lineage documentation gaps across connections
- Data engineers deciding where to prioritize lineage remediation work
- Platform/catalog admins reporting lineage health by connection and schema
- Stakeholders evaluating which assets are safe to build downstream impact analysis on
What you receive
How the recipe works (guided flow)
Step 1 — Build Active Asset Inventory
Compiles every active table and column in scope as the full population, so no asset is ever silently dropped before analysis begins.
Step 2 — Resolve Column-Level Lineage Edges
Flattens raw column-to-column lineage records into resolved source-table-to-destination-table edges for analysis.
Step 3 — Classify Lineage Presence
Tags every asset as Connected, Upstream Only, Downstream Only, or Complete Orphan based on whether lineage edges exist.
Step 4 — Calculate Column Lineage Coverage
Measures what percentage of each table's active columns actually participate in a lineage edge, turning "has lineage" into a real coverage number.
Step 5 — Trace Lineage Depth Backward
Walks backward from each table up to four hops to determine how far its data can be traced to a source, with built-in cycle detection to keep results accurate.
Step 6 — Determine Lineage Readiness Signal
Combines presence, coverage, and depth into one readiness label per asset, along with a plain-English explanation of any gap.
Step 7 — Summarize and Visualize Results
Rolls up readiness and depth results into summary tables and charts by connection, schema, and asset type for quick reporting.
Sample insights (illustrative)
| Insight Category | What the recipe discovered | Business Impact |
|---|---|---|
| Orphan Detection | 18% of active tables in the Snowflake_Prod connection have zero recorded lineage | Flags previously invisible blind spots before any impact analysis can rely on this data |
| Column Coverage | A "Connected" table shows only 32% column-level lineage coverage despite appearing fully linked | Prevents false confidence in tables that look connected but are largely untraceable |
| Depth & Cycle Detection | A finance reporting table traces back 3 hops to its source, with one circular path safely detected and excluded | Gives an accurate depth measure teams can trust instead of an inflated or misleading trace |
Make sure the following ingredients are available in your workspace:
- Connected datasets are crawled and profiled
- Columns have distinct counts and top values generated